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Abstract — Wyner-Ziv coding (WZC) is a compression tech- 
nique using decoder side information, which is unknown at the 
encoder, to help the reconstruction. In this paper, we propose and 
implement a new WZC structure, called residual WZC, for the 
quadratic-Gaussian Wyner-Ziv problem where side information 
can be arbitrarily distributed. In our two-stage residual WZC, the 
source is quantized twice and the input of the second stage is the 
quantization error (residue) of the first stage. The codebook of the 
first stage quantizer must be simultaneously good for source and 
channel coding, since it also acts as a channel code at the decoder. 
Stemming from the non-ideal quantization at the encoder, a prob- 
lem of channel decoding beyond capacity is identified and solved 
when we design the practical decoder. Moreover, by using the 
modified reinforced belief-propagation quantization algorithm, 
the low-density parity check code (LDPC), whose edge degree is 
optimized for channel coding, also performs well as a source code. 
We then implement the residual WZC by an LDPC and a low- 
density generator matrix code (LDGM). The simulation results 
show that our practical construction approaches the Wyner- 
Ziv bound. Compared with previous works, our construction 
can offer more design flexibility in terms of distribution of side 
information and practical code rate selection. Q 

I. Introduction 

Wyner-Ziv coding (WZC) [1], which is a generalization 
of the lossless Slepian-Wolf coding (SWC) [2], refers to a 
lossy source coding with side information at the decoder only. 
Among the applications of WZC [3], one important scenario is 
the quadratic-Gaussian problem, where the difference between 
the source and the side information is Gaussian and mean- 
square error (MSE) is adopted as the distortion measure. 
For this case, WZC does not incur rate loss when compared 
with source coding with side information available at both 
the encoder and the decoder. This zero-rate-loss result was 
originally proved for the case where the side information 
is Gaussian [1], but recently generalized to the case where 
the side information can be arbitrarily distributed [4]. There 
are many applications of quadratic-Gaussian WZC, such as 
distributed source coding in the sensor networks [3] and the 
protocols in the relay networks [5]. 

The theoretical analysis of Wyner and Ziv [1] was based 
on the non-structured codebook which is hard to implement 
in practice. Later on, some theoretical works based on struc- 
tured codebooks were proposed [4] [6] [7]. However, only 
[4] focused on the quadratic-Gaussian cases. Whether the 
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theoretical results in [6] [7] can be applied to such cases 
is unknown. Moreover, the results in [4] [6] [7] require two 
nested codebooks good for the source and/or channel coding. 
Practically constructing such good codebooks with nested 
structure in [4] [6] [7] is still a challenging task. 

Instead of building WZC from nested codebooks as in [4] 
[6] [7], we proposed a new coding structure in our previous 
work [8], where two codebooks without nested structure were 
used in a two-stage serial quantization process. The encoder 
first quantizes the source once, and then uses another codebook 
to quantize the quantization error of the first stage. The 
quantization index of the second stage is then sent to the 
decoder. We name the coding in [8] residual WZC, which 
reflects some resemblances of our coding structure to the 
residual vector quantizer [9, Sec 12.11] for source coding 
without side information. In a theoretical random coding 
setting, the residual WZC was proved to be Wyner-Ziv-bound 
achieving. 

In this paper, we show that using graph-based codes, 
the theoretically-optimal residual WZC in [8] can be prac- 
tically implemented. Different from the celebrated quadratic- 
Gaussian WZC in [10] [11], our scheme can approach the 
Wyner-Ziv bound for all rate regimes when the side infor- 
mation is arbitrarily distributed. By using the loss to the 
Wyner-Ziv bound as the performance metric, simulation shows 
that the performance of our code is comparable to that in 
[10]. However, in contrast to [10], our simulation result is 
independent of the distribution of the side information, as long 
as the variance of the side information is the same. Besides, 
our WZC has a complexity similar to that of [10] and linear in 
the codeword length, and much lower than that of the lattice 
decoder (NP problem) in [4]. 

Practically implementing the residual WZC is not trivial. 
Firstly, the codebook of the first stage quantizer acts as a 
channel code in our decoder, and must be simultaneously 
good for source and channel coding (SSC). Moreover, the 
non-ideal practical quantizers at the encoder make the channel 
decoder operate in a rate regime above capacity. We propose a 
method which can increase the equivalent signal-to-noise ratio 
(SNR) at the channel decoder to solve this problem. Secondly, 
although the low-density parity check code (LDPC) has been 
proved to be SSC theoretically [12], the edge degree design 
and practical quantization algorithm for SSC LDPC are still 
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Coding structure of the proposed residual Wyner-Ziv coding 



unknown. We modify the recently proposed reinforced belief 
propagation (RBP) algorithm [13] so that the LDPC, whose 
edge degree is originally optimized for channel coding, is 
good for source coding. The codebook of the second stage 
of our residual WZC is chosen as a low-density generator 
matrix code (LDGM). We also apply the RBP algorithm to 
LDGM, which is much simpler than the well-known hard- 
decimation-based algorithm [14] for LDGM, and reduces the 
overall WZC complexity a lot. More detailed comparisons 
between our practical construction and previous works can 
be found in Section HV-BI 

II. System model and residual Wyner-Ziv coding 

In the considered quadratic-Gaussian Wyner-Ziv problem, 
the relationship between the length-n source vector x and the 
arbitrary side information y„ is □ 



x = y a + v, 



(1) 



where v is an independent and identically distributed (i.i.d.) 
Gaussian vector with each element distributed as N(0,Pv), 
and v is independent of y a . Moreover, y a can be arbitrarily 
distributed. Denoting the reconstruction as x, the "quadratic" 
MSE distortion measure is adopted with maximum distortion 

D 

(2) 



E[(x-x) 2 ] <D 



The Wyner-Ziv bound is then [4] 

% z (73) = ilog(^ 



(3) 



when P v > D. When P v < D, R WZ (D) = and we will neglect 
this trivial case in the following. 

A. Structure of the proposed residual Wyner-Ziv coding 

We briefly review the residual WZC in [8], and the coding 
structure is depicted in Fig. Q] First, we introduce some 
definitions as follows 



*In this paper, a vector is denoted in bold lower-case, while the superscript 
T denotes the transpose of a vector. A zero-mean Gaussian random variable 
with variance o" 2 is denoted by N(0.G 2 ). A random variable X for Shannon's 
random coding setting is denoted in capital italic. The entropy is denoted as 
h(-). All logarithms are of base 2. 



Definition 1: Given a vector x = (jci, . . . ,x n ) T and a prede- 
termined value A, the modulo A operation is defined as 

x mod A = (xi -Qa(xi),...,x„ -Qa{x„)) T 7 

where Qa{x{) is the nearest multiple of A to X{, Vi. 

Definition 2 (mod- A distance): The mod-A distance be- 
tween two vectors a = (a\, . . . ,a n ) T and b = {b\,... ,b„) T is 

||b-a||i = £ [(bi-ai) mod A] 2 . 

i=l 

The residual WZC is described in detail as follows: 
Encoder part: The input of the first-stage quantizer C\ in 
Fig. Q] is 

(ocx + d) mod A, 

where a is a scaling factor which will be determined later, d is 
a randomly generated dither signal known both at the encoder 
and the decoder, and A is called as the modulo size. The 
entries of the dither are uniformly distributed in the interval 
[—§,4]. According to Crypto Lemma in [8] [4], we know 
that (ax + d) mod A is uniform in the region [— 4,§]" and 
independent of x. Then the distribution of x, determined by 
the distribution of side information y a from (Q~|i, will not affect 
the quantization result of our residual WZC encoder. The 
first-stage quantizer Ci searches a codeword — ci in Ci such 
that the mod-A distance (in Defintion [2]) between ax + d and 
— Ci is minimized, under the distortion constraint ^E[ei r ei] < 
ocZ) + <x 2 Pv ■ The quantization error after the first stage is 



ei = (ocx + d + ci) mod A. 



(4) 



The input of the second-stage quantizer C2 in Fig. Q] is the 
quantization error ei of the first stage in The distortion 
constraint for the second stage is -E^ 7 ^] < aD, and the 
quantization output and the quantization error of the second 
stage are C2 and 

e 2 = (ei -C2) mod A, (5) 

respectively. Finally, the encoder sends the index representing 
C2 to the decoder. 

Decoder part: The decoder receives the index representing 
C2 and the side information y a . As in Fig. Q] the decoder first 
computes w = (C2 — OCy„ — d) mod A. From [8], equivalently 
we have 

w = (ci + av — e2) mod A. (6) 

Then we can channel decode Ci from w, by treating 

av - e 2 (7) 

as the equivalent channel noise, where e2 is given in (0. By 
denoting the channel decoder output as £1, we can compute 

v, the reconstruction of v, as \ = (w — Ci) mod A. Finally, the 

reconstruction x is 

x = y„ + v. 

Let the code rates of C\ and C2 be R\ and Ro, respectively. 
According to [8], by letting a be 



a= l-D/Py, 



(8) 
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Graph-based code implementation of residual Wyner-Ziv 

CODING 

there exists codebooks C\ and C 2 such that the MSE distortion 
constraint (O is met if 

Ri >logA-ilog(27i e (aPv)) + ei,/?2>^log(^) + e 2 (9) 

Ri <logA-ilog(27te(a/V)) + ei, (10) 

where £1 , £2 — > as A — > °° and the code length 77 — > °°. Indeed 
(O ensures the success of the two quantization processes in the 
encoder, while ( fTOb ensures the success of channel decoding 
in the decoder [8]. Then from (l9l(fT0li. we know that one can 
achieve the Wyner-Ziv bound in (0 as 



R 2 



1 ^ ( Pv \ 
2 l0g( D } ' 



(11) 



III. Design flow for graph-based code 

IMPLEMENTATION 



Theoretically, from [8], the C\ must be a SSC code while C 2 
must be a good source code. We implement C\ and C 2 in Fig.Q] 
by an LDPC and an LDGM, respectively. The detailed block 
diagram of our practical implementation is shown in Fig. |2] 
Before introducing our flow to select the code parameters and 
the detailed encoding/decoding algorithms in Section IIII-BI 
in Section UlI-AI we first discuss some problems encountered 
when building our flow for the practical implementation. 
A. Issues for practical implementation 

Finite modulo size : In [8], the Wyner-Ziv bound is achieved 
while the modulo size A approaches infinity. However, infinite 
modulo size A is impractical since the rate R\ in ([9) will 
become infinite, too. Thus we propose the following method 
to estimate how large A should be in practice. From [8], the 
£1 and £2 in (O satisfy 



£2 <£i = h{aV - E 2 ) - h((aV - E 2 ) modA), 



(12) 



where V ~ N(0,Pv), and £2 is a random variable correspond- 
ing to e 2 in (0 in the Shannon-random coding setting. The 
distribution of E 2 is detailed in [8]. By numerically calculating 
the right-hand-side (RHS) of the equality in (fl2l) . we can 
choose A to make £1 and £2 smaller than a threshold £. 



In the following, we use the modulo size A E to represent 
the sufficiently large modulo size such that we can essentially 
neglect £1 and £2 in 

Channel decoding beyond capacity : With A £ obtained 
previously, we can select R\ from (O by replacing A with A £ 
and neglect £1. However, if we directly implement the channel 
decoder in Fig. [2] with selected R\ and A £ , the channel decoder 
will always fail. The reason is as follows. Since no practical 
quantizers C\ and C 2 can exactly achieve the rate-distortion 
bound, we will get the practical quantization error variance 
£>2.e of e2 in (0 larger than the theoretical one predicted in 
[8]. From (0, the variance of the equivalent noise (0 will also 
be larger than the theoretical value in [8], which makes the 
selected R\ operating in the regime above the channel capacity 
in ( flOt with A = A £ . 

To solve the channel decoding beyond capacity problem 
described above, we propose a method which increases A e 
to A p while Ri is fixed as that in Step 2 of Table [Q The 
key idea is that the equivalent SNR at the channel decoder in 
Fig. [2] will also increase. To be more specific, from the random 
codebook construction in [8], we know that the optimal 
constellation of C\ is uniformly distributed in [— -j, -^]. Thus 
we use the uniform pulse amplitude modulation (PAM) as the 
constellation of C\, and the equivalent SNR of the channel (0 
with A = A £ is 

A\/k 



SNR e = 



(13) 



where I is a constant related to the order of the PAM 
constellation. If we increase the modulo size from A £ to 
A p with rates R\ and R 2 unchanged, the quantization error 
variance of e2 in (0 will become 



D 2 , P = (A P /A £ ) 2 D 2 , 



(14) 



because the variance of Ci's input, which is uniformly- 
distributed, will increase from A £ /12 to A 2 / 12. Then the 
equivalent SNR becomes 



Aj/k 



A 2 Jk 



SNR,, = - 

' D 2 , p + a 2 Pv D 2 , £ + (^) 2 a 2 /V 

which is larger than SNR £ in dT3l l since A p > A £ . 

Now we consider the loss of the practical channel decoder 
with practical C\ in Fig. |2] compared to the optimal channel 
coding in [8]. For the practical channel coding, let o„ £ be 
the variance of the maximum tolerable equivalent noise for 
the successful decoding, where the PAM constellation of C\ 
is chosen according to A £ . Then under the non-ideal channel 
decoding, we have 



a 2 P v , 



(15) 



where the RHS is the variance of the equivalent noise Q. 
When the PAM constellation of C\ is chosen according to 
Ap instead of A £ , the signal power of codeword Ci is scaled 
by (Ap/A £ ) 2 . Then the maximum tolerable equivalent noise 



variance a 2 , p for the practical channel coding can be estimated 



by 



TABLE I 
Design Flow of Residual WZC 



°L = 0V A e) 2 °il 



(16) 



Now we wish a 2 _ > Z)2, p + OC 2 Py to ensure the practical 
channel decoding in (O being successful. Using this criterion 
with ( fl4l i and ( fl6] l, the practical modulo size A p in Fig.|2]must 
meet, 



A P >( 



— § _ )i 



(17) 



O 2 £ and D2, £ can be obtained via numerical simulations. From 
( [T5| >. we know that A ; , selected according to (fTTT i is lager than 
A E . Although increasing the modulo size can solve the channel 
decoding beyond capacity problem, there will be a loss in the 
final distortion compared with the theoretically predicted D 
using R2 in ( fTTT i owing to the increment of the quantization 
error variance of e.2 in © from ( TBI . 

Edge degree for SSC LDPC : One of the critical parts 
of our residual WZC is the requirement of a practical SSC 
code C\. The best known SSC so far is the trellis coded 
quantation (TCQ)/modulation. However, according to the sim- 
ulation results in [15], using TCQ as our C\ will result in 
significant performance loss compared with the Wyner-Ziv 
bound. Alternatively, in [12], LDPC was proved to be an 
SSC code for the binary source/channel under the optimal en- 
coder/decoder. However, the theoretical proof in [12] gives no 
hints for the edge degree design and the practical quantization 
algorithms for SSC LDPC. Recently, the RBP algorithm was 
proposed in [13] for the quantization of the binary source with 
LDPC. However, the LDPC in [13] is not SSC, since its edge 
degree exhibits the ultra-sparse structure which makes LDPC 
poor for the channel coding when the belief -propagation (BP) 
algorithm [16] is used. 

To solve the practical design problem for SSC LDPC, we 
propose using LDPC with the edge degree optimized for the 
channel coding, and modifying the binary RBP in [13] for 
our continuous LDPC quantization with mod-A operation. 
Although our edge degree for the first-stage quantizer C\ in 
Fig. 12 is sub-optimal for the source coding, our simulation 
results show that it suffices to make the overall residual WZC 
approach the Wyner-Ziv bound. Indeed, our simulation results 
in Section IIV-AI show that our LDPC has a shaping loss [9] 
about only 0.5 dB away from the rate-distortion bound when 
the quantizer input is uniformly-distributed. Our modified RBP 
algorithm is given in Sec. IIII-BI 
B. Design flow and encoding/decoding algorithms 

Our overall design flow is summarized in Table UJ and is 
explained in detail as follows. The Step 1 and 2 in Tableware 
described previously in Sec. IIII-AI To design the edge degree 
of the SSC LDPC in Step 3, as discussed in Sec. IIII-AI we 
adopt the EXIT chart fitting approach in [17] to obtain an 
LDPC good for the channel coding. By running the BP channel 
decoding algorithm, we can calculate the a 2 £ described before 
< Tl5T > for our LDPC with the PAM constellation points. For Step 
4, the rate of the LDGM is given in (fTTT i. and the quantization 



1: Determine the target (ideal) MSE distortion D: 

Given Py and WZC rate R2, determine D from Jilt 
2: Determine code rate R{ of the lst-stage quantizer: 

Given E, find A = A e such that £j < 8 from | |121 . 

Compute R{ from l(9j with A =A E . 
3: Find the maximum tolerable noise variance of the channel 

decoder with A — A e : 

Design SSC LDPC with rate Ri. Calculate CT^ e with BP. 
4: Find the distortion D2.E of the 2nd-stage C2 with A =A E 
Design LDGM with rate R2. 

Calculate D 2 . e with modified RBP applied to LDPC and LDGM. 
5: Determine the practical modulo size A p : 

Calculate A p using the RHS of (TTJ. If the LDPC decoder fails, 



alphabet is uniformly spaced with energy a 2 Pv suggested in 
[8]. The edge degree of LDGM is obtained similarly as in 
[18]. 

To calculate Z)2, £ in Step 4 of TableU we modify the RBP in 
[13] as the quantization algorithm for both LDPC and LDGM 
at the WZC encoder in Fig. |2] The RBP is a generalization 
of BP by adding an reinforcement term controlled by the 
constants yo, Yi £ [0, 1] to the marginal L-value calculated from 
the variable nodes (VND) of the graph-based code. The main 
modification of our RBP algorithm is the a priori information 
from the source. Taking quantizer Ci as an example. We let 
Uj be the ith element of Ci's input (uniform), and c\ j be the 
/th coded symbol of codeword ci. To reflect the modulo A e 
operation before C\ in Fig.Q] we use the following conditional 
probability density function (PDF) to calculate the a priori 
information 



(18) 



where f Ga 2 (g) is the PDF of N(0,<3 2 £ ), and Z is the integer 
set. The res't E of the algorithm is the same as that in [13] and 
is omitted here. After obtaining Z)2, £ in Step 4, we can follow 
Step 5, which is described in Sec. IIII-AI to get the practical 
modulo size A p in Fig. [2] and complete our design. 

Finally, note that the complexity of the BP channel decoding 
algorithm for our WZC decoder is linear [16] in codelength 
(i.e. 0(n)), also is the RBP algorithm used in our WZC 
encoder. Adopting the RBP algorithm instead of the 0(n 2 ) 
hard-decimation-based one in [14] [18] significantly reduces 
our computation complexity. 

IV. Design example and Discussions 

In our design example, different from the Gaussian side 
information in [10] [19], we let each element of the side 
information y in (H) be uniformly distributed in [—A/2, A/2]. 
Due to the dither, the distribution of y will not affect our 
performance. The details of our design example is given in 
Sec. IIV-AI with the distortion performance given in the end of 
this subsection. Finally, Sec. II V-B1 provides more discussions 
on our work. 



A. Design example with detailed code parameters selection 

Following our design flow in Table |U we first set Py in 
© as 0.28 and the WZC rate R 2 = 0.953, and then the ideal 
distortion D is 0.0747. For Step 2, given £=0.005, we find 
that choosing A E as 3 is sufficient. R\ is 0.68 bpcu. For Step 
3, we use 2-PAM LDPC to implement C\ with constellation 
points ±4p. To achieve the WZC bound, we choose codeword 
length n = 10 s symbols per source block. The degree profile 
is : check nodes (CND): 100% of degree 12; VND: 35.36% 
of degree 2, 44.74% of degree 3, and 19.89% of degree 9. By 
applying the BP channel decoding algorithm, we obtain aj l£ 
as 0.185. 

For Step 4, we adopt 4-ary LDGM where every two bits 
of the LDGM CND are Gray-mapped to a 4-ary symbol. 
The degree profile can be found in [18]. By running the 
RBP quantization algorithm with y = 1, Yl = 0.99980 at the 
LDPC and LDGM, we obtain D 2 , £ as 0.0577. The Monte 
Carlo simulation tests 2000 blocks of uniformly-distributed 
source samples and 3000 iterations are run for each source 
block. The summation over integer set Z in (fL8b is obtained 
by limiting \b\ < 3,b 6 Z. As in [13], in some few cases the 
RBP algorithm diverges, i.e. not all the constraints of CND 
are satisfied. For these few cases, as in [13], restarting the 
RBP by adding 0.00001 to yi will solve the problem. Finally, 
from Step 5, we calculate the lower-bound of A p in $1% as 
3.261. The bit error rate of LDPC is smaller than 10~ 4 when 
the modulo size A p is 3.29. 

Finally, we run the Monte Carlo simulation to test 2000 
source blocks. The distortion loss compared with the ideal D 
is 0.995 dB at the WZC rate 0.9531 b/s. The best simulation 
result known with Py = 0.28 [10], which is only for the 
Gaussian side information, has 1.07 dB loss at the rate 1.07 
b/s and block length n = 10 5 . The results in [10] are obtained 
using 8192-state trellis coded quantization (TCQ) and 3-stage 
SWC (three LDPCs in the SWC encoder). Our work is as 
competitive as [10]. Besides, our SSC LDPC has only 0.48 dB 
SNR loss compared to the capacity and 0.43 dB shaping loss 
compared to the rate-distortion bound. More design examples 
can be found in [20]. 

B. Detailed comparison 

Although the well-known practical WZC design [10] also 
performs a two-stage quantization/cpmpression process, the 
second stage quantizer/compressor in [10] is lossless with 
input being the quantization output of the first stage. In our 
residual WZC, the second stage quantizer is lossy with input 
being the quantization error of the first stage, and thus the 
decoder structure is also different from that in [10]. Compared 
with the WZC in [10], the main advantage of our coding 
scheme is the flexibility. Firstly, as shown in [8], our residual 
WZC can approach the Wyner-Ziv bound with arbitrary side 
information in all rate regimes, while the WZC in [10] [11] 
can only guarantee the optimality in high rate regimes. In 
[10], the lower the WZC rate the more severe the rate loss 
is. Secondly, our construction offers more flexibility in the 
practical code rate selection. In [10], the WZC is implemented 



by a rate Rn TCQ and a multilevel SWC (formed by Rn 
LDPCs). The rate of TCQ Rn is limited to be an integer, 
and for each Rn = 1,2,..., only one possible WZC rate can 
be implemented. However, almost all the rational WZC rates 
can be implemented by our construction. 

Our construction is also different from and more flexible 
than the graph-based construction in [6] for which the encoder 
codebook is nested. Moreover, the WZC in [6] has only been 
proven to be optimal for the binary symmetric source with 
Hamming distortion measure. Whether it can be extended 
to the quadratic-Gaussian case as considered in this paper 
or not is still unknown. Finally, our simulation results show 
that LDPC itself suffices to be a good SSC code, thus the 
compound LDGM/LDPC construction proposed in [6] may 
not be necessary for our first-stage quantizer C\ . 



V. Conclusion 

In this paper, we considered the quadratic-Gaussian Wyner- 
Ziv problem where side information can be arbitrarily dis- 
tributed. We implemented the theoretically-claimed residual 
WZC by LDPC and LDGM. We identified and solved a 
problem called the channel decoding beyond capacity problem 
when designing our practical decoder. Moreover, we modified 
the RBP algorithm to make the LDPC with the edge degree 
optimized for the channel coding perform well as a source 
code. The simulation results showed that our practical con- 
struction approaches the Wyner-Ziv bound, and has a similar 
performance compared with previous works. Moreover, our 
construction can offer more design flexibility in terms of the 
distribution of the side information and the practical code rate 
selection. 
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